test: Deflake L0_socket reuse-port request distribution check - #9001

What does the PR do?

L0_socket starts 3 servers sharing the HTTP/gRPC port (--reuse-*-port=true), sends a single batch of 11 client processes, and fails if any server's nv_inference_request_success is 0. The kernel uses a hash to pick which SO_REUSEPORT listener gets each new connection, so with 11 connections there's roughly a 3·(2/3)^11 ≈ 5% chance per protocol that one server gets nothing. That matches the intermittent pseudo-nightly failures: gRPC counts were 0/16/6 on 10-06 and 8/14/0 on 10-02, and the runs in between passed.

Change: send batches of 11 clients until all three servers have served at least one request, up to MAX_CLIENT_ROUNDS=3. If a server still has 0 after 3 rounds, the test fails as before. A listener that really never receives connections is still caught; only the chance-based miss goes away. The test also prints the per-round counts and waits on each client PID individually. Before, wait $pids only returned the exit status of the last client, so failures in other clients went unnoticed.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging.
• All template sections are filled out.

Commit Type:

• test

Related PRs:

None

Where should the reviewer start?

qa/L0_socket/test.sh, the reuse-port distribution loop near the end of the file.

Test plan:

• bash -n and pre-commit pass. I tested the loop logic locally with a fake client that assigns requests at random. I haven't run it against a real server. It needs an L0_socket--base CI run.
• CI Pipeline ID:
  75864706
  75864846
  75865062
  75865453
  75867062
  75890914
  75890967
  75891050
  75891151
  75891185
  75891245
  75891343
  75891394
  75891433
  75892802
  75892855
  75893129
  75893186
  75893251
  75893324
  75893385
  75893444

Caveats:

In the worst case the test takes up to 3x as many client runs. In practice it almost always stops after round 1.

Background

The pseudo-nightly CI health report flagged L0_socket--base as newly broken (dl/dgx/tritonserver job 473179531).

Related Issues:

• Relates to the pseudo-nightly L0_socket--base flake (no ticket yet)

Link to Devin session: https://nvidia-cloud.devinenterprise.com/sessions/42b5577b240344db884012e785abfcc6
Open in Devin Desktop: https://nvidia-cloud.devinenterprise.com/desktop/session/42b5577b240344db884012e785abfcc6?variant=devin